{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Setting Up AIStore on Google Colab\n",
        "\n",
        "This notebook guides you through installing and running [AIStore](https://github.com/NVIDIA/aistore) in Google Colab. You'll also learn how to configure optional AWS and GCP backends with AIStore for seamless cloud data integration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a href=\"https://colab.research.google.com/github/NVIDIA/aistore/blob/main/python/examples/google_colab/aistore_deployment.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 1: Install Go (required for AIStore)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Install Go (v1.23.1)\n",
        "!wget https://go.dev/dl/go1.23.1.linux-amd64.tar.gz\n",
        "!sudo tar -C /usr/local -xzf go1.23.1.linux-amd64.tar.gz\n",
        "!mkdir go\n",
        "\n",
        "# Add Go to environment variables\n",
        "import os\n",
        "os.environ['GOPATH'] = '/content/go'\n",
        "os.environ['PATH'] += ':/usr/local/go/bin:/content/go/bin'\n",
        "\n",
        "# Check Go version to ensure installation\n",
        "!go version"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 2: [Optional] Configure AWS Backend\n",
        "Skip this section if you do not want to configure AWS integration.\n",
        "You will need to enter your AWS credentials below to enable AWS as a backend for AIStore."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create the .aws directory and config file with AWS credentials\n",
        "! mkdir -p ~/.aws\n",
        "\n",
        "# Add your AWS region, access key, and secret key\n",
        "! echo -e \"[default]\\nregion=us-east-2\\naws_access_key_id=ACCESS_KEY_ID\\naws_secret_access_key=SECRET_ACCESS_KEY\" > ~/.aws/config"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 3: [Optional] Configure GCP Backend\n",
        "Skip this section if you do not want to configure GCP integration.\n",
        "You'll need to enter your GCP credentials below to enable GCP as a backend for AIStore."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Create GCP credentials file\n",
        "os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/content/gcp_creds.json'\n",
        "\n",
        "# Define your GCP credentials JSON here (replace values)\n",
        "gcp_creds_json = '''{\n",
        "  \"type\": \"service_account\",\n",
        "  \"project_id\": \"PROJECT_ID\",\n",
        "  \"private_key_id\": \"PRIVATE_KEY_ID\",\n",
        "  \"private_key\": \"-----BEGIN PRIVATE KEY-----\\\\PRIVATE_KEY\\\\n-----END PRIVATE KEY-----\\\\n\",\n",
        "  \"client_email\": \"xxx@yyy.iam.gserviceaccount.com\",\n",
        "  \"client_id\": \"CLIENT_ID\",\n",
        "  \"auth_uri\": \"https://accounts.google.com/o/oauth2/auth\",\n",
        "  \"token_uri\": \"https://oauth2.googleapis.com/token\",\n",
        "  \"auth_provider_x509_cert_url\": \"https://www.googleapis.com/oauth2/v1/certs\",\n",
        "  \"client_x509_cert_url\": \"https://www.googleapis.com/robot/v1/metadata/x509/CERT_URL\"\n",
        "}'''\n",
        "\n",
        "# Write the GCP credentials to a file\n",
        "with open('/content/gcp_creds.json', 'w') as json_file:\n",
        "    json_file.write(gcp_creds_json)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 4: Clone AIStore repository"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Clone the AIStore GitHub repository\n",
        "! git clone https://github.com/NVIDIA/aistore.git"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 5: Deploy AIStore with AWS and GCP Backends\n",
        "In this step, AIStore is deployed with 1 target, 1 proxy, and 3 mountpaths, including both AWS and GCP backends.\n",
        "To exclude AWS or GCP, remove the respective flags (`--aws` or `--gcp`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Clean up any existing deployment and start AIStore\n",
        "! make -C aistore cli kill clean\n",
        "! nohup bash -c \"PORT=51080 ./aistore/scripts/clean_deploy.sh --target-cnt 1 --proxy-cnt 1 --mountpath-cnt 3 --aws --gcp\" > output.log 2>&1 &"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 6: Check AIStore Logs\n",
        "You can view the AIStore deployment logs using the command below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Check the deployment logs\n",
        "! cat output.log"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 7: Verify AIStore Cluster is Running\n",
        "Check if the AIStore cluster is reachable and healthy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Check if the cluster is reachable and healthy\n",
        "! curl -i http://localhost:51080/v1/health"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 8: Install AIStore Python SDK and Connect to Cluster\n",
        "This step installs the AIStore Python SDK and connects to the running AIStore cluster."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Install AIStore Python SDK\n",
        "! pip install aistore -q\n",
        "\n",
        "from aistore import Client\n",
        "\n",
        "# Initialize the AIStore client\n",
        "ais_client = Client(\"http://localhost:51080\")\n",
        "ais_client.cluster().get_info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Step 9: Display AIStore Cluster Information\n",
        "You can use the following command to display detailed information about the AIStore cluster."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Show detailed AIStore cluster information\n",
        "! AIS_ENDPOINT=http://localhost:51080 ais show cluster"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
